Papers with VLP models
KG-FLIP: Knowledge-guided Fashion-domain Language-Image Pre-training for E-commerce (2023.acl-industry)
Copied to clipboard
| Challenge: | Various visionlanguage pre-training (VLP) models learn cross-modal alignment from large-scale well-aligned image-text datasets without leveraging external knowledge. |
| Approach: | They propose a knowledge-guided fashion-domain language-image pre-training framework that learns fine-grained representations in e-commerce domain and utilizes external knowledge to improve the pre-train efficiency. |
| Outcome: | The proposed framework outperforms state-of-the-art models on Amazon and Fashion-Gen datasets by large margins. |
Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack (2025.findings-naacl)
Copied to clipboard
| Challenge: | Visual-Language Pre-training (VLP) models are vulnerable to adversarial examples . previous studies have focused on improving adversariality of models . |
| Approach: | They propose a local shuffle and sample-based attack that randomly shufts one of the local image blocks and generates adversarial images and samples around them. |
| Outcome: | The proposed attack outperforms other advanced attacks on Large Vision-Language Models and outperformed previous attacks on Visual-Langue Pre-training models. |
FashionKLIP: Enhancing E-Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph (2023.acl-industry)
Copied to clipboard
Xiaodan Wang, Chengyu Wang, Lei Li, Zhixu Li, Ben Chen, Linbo Jin, Jun Huang, Yanghua Xiao, Ming Gao
| Challenge: | Recent advances in visual-language pre-trained (VLP) models have greatly improved cross-modal retrieval performance . however, the fine-grained interactions between objects from different modalities are far from well-established . e-commerce domain lacks sufficient training data and fine-granular cross-modulal knowledge . |
| Approach: | They propose a visual-language pre-trained (VLP) image-text retrieval model that integrates cross-modal knowledge into the model to improve performance. |
| Outcome: | The proposed model improves performance on e-commerce image-text retrieval task by a large margin. |
MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large-scale visual-language pretraining models have shown remarkable capabilities in understanding both vision and language. |
| Approach: | They propose a multi-teacher cross-modality alignment distillation technique to integrate the advantages of single-stream and dual-stream models. |
| Outcome: | The proposed model is lightweight and has only 100M running memory and 8.0ms search latency. |
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2023.eacl-main)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-trained (VLP) models generate unfaithful or nonsensical texts given the source input, which is called hallucination. |
| Approach: | They propose a VLP loss-based model to mitigate object hallucination by decoupling VLP objectives and a token-level image-text alignment. |
| Outcome: | The proposed model reduces object hallucination by 17.4% on two benchmarks. |
TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-and-language pre-training models suffer from long visual sequences . experimental results show that TRIPS gains a speedup of 40% over previous similar VLP models . |
| Approach: | They propose an efficient vision-and-language pre-training model with text-relevant image patch selection, TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference. |
| Outcome: | The proposed model can speed up training and inference by 40% over previous models. |
Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language pretraining (VLP) lack effectiveness and efficiency in downstream multimodal tasks. |
| Approach: | They propose a flexible vision-and-language pre-training model by incorporating cross-modal fusions into a dual-encoder architecture and a cross-module knowledge transfer strategy to guide the training process. |
| Outcome: | The proposed model is well-equipped with effectiveness and efficiency compared with other strong VLP models. |
Probing Cross-modal Semantics Alignment Capability from the Textual Perspective (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In recent years, vision and language pre-training (VLP) models have advanced the state-of-the-art results in a variety of cross-modal downstream tasks. |
| Approach: | They propose a new probing method that is based on image captioning to first empirically study the cross-modal semantics alignment of VLP models. |
| Outcome: | The proposed method analyzes captions generated by five popular VLP models to reveal how well they align with visual words and how well these align with images. |
APoLLo : Unified Adapter and Prompt Learning for Vision Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | APoLLo improves generalization capabilities of vision-language pretrained models . despite being largely successful in terms of generalization, these models are difficult to fine-tune for few-shot learning-based downstream tasks. |
| Approach: | They propose a multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. |
| Outcome: | The proposed approach improves generalization capabilities of vision-language pretrained models . it achieves a relative gain of 6.03% over MaPLe on 10 diverse datasets . |
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances on self-supervised learning have led to powerful vision-language pre-training models that achieve state-of-the-art performance on a wide range of cross-modal tasks. |
| Approach: | They propose a vision-language pre-training framework that reformulates discretized object positions and language in a unified language modeling framework. |
| Outcome: | The proposed model improves performance on position-sensitive vision-language (VL) tasks and also improves on position insensitive tasks. |
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks. |
| Approach: | They propose a new framework for a semantically-aware contrastive learning that minimizes the MI between false negative and positive samples . |
| Outcome: | The proposed framework minimizes the MI between false negative samples and positive samples even though they share similar semantics. |
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling. |
| Approach: | They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space. |
| Outcome: | The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks. |